[AMORO-1906]Refactor code of "Table" and "Catalog" to support Paimon format integration #1960

shidayang · 2023-09-12T12:40:35Z

Why are the changes needed?

Close #1906.

Brief change log

Replace all ArcticTable with AmoroTable.
Refactor 'orphan file clean' and 'snapshot expire'

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

…copy # Conflicts: # ams/server/src/main/java/com/netease/arctic/server/table/executor/OrphanFilesCleaningExecutor.java # ams/server/src/main/java/com/netease/arctic/server/table/executor/SnapshotsExpiringExecutor.java # ams/server/src/test/java/com/netease/arctic/server/table/executor/TestSnapshotExpire.java

codecov · 2023-09-13T02:03:02Z

Codecov Report

Attention: 458 lines in your changes are missing coverage. Please review.

Files	Coverage Δ
...etease/arctic/server/DefaultOptimizingService.java	`22.15% <ø> (ø)`
.../netease/arctic/server/catalog/CatalogBuilder.java	`43.75% <100.00%> (+10.41%)`	⬆️
...ease/arctic/server/catalog/IcebergCatalogImpl.java	`73.33% <100.00%> (+27.17%)`	⬆️
...tic/server/catalog/InternalIcebergCatalogImpl.java	`89.28% <100.00%> (+0.39%)`	⬆️
...arctic/server/catalog/MixedIcebergCatalogImpl.java	`73.68% <100.00%> (+1.46%)`	⬆️
...m/netease/arctic/server/catalog/ServerCatalog.java	`100.00% <ø> (ø)`
...ease/arctic/server/optimizing/OptimizingQueue.java	`73.81% <100.00%> (+0.14%)`	⬆️
...tease/arctic/server/table/DefaultTableService.java	`85.82% <100.00%> (-0.77%)`	⬇️
.../com/netease/arctic/server/table/TableManager.java	`0.00% <ø> (ø)`
.../com/netease/arctic/server/table/TableRuntime.java	`74.50% <100.00%> (+0.41%)`	⬆️
... and 34 more

... and 33 files with indirect coverage changes

📢 Thoughts on this report? Let us know!.

ams/server/src/main/java/com/netease/arctic/server/catalog/InternalIcebergCatalogImpl.java

ams/server/src/main/java/com/netease/arctic/server/dashboard/ServerTableDescriptor.java

ams/server/src/main/java/com/netease/arctic/server/optimizing/OptimizingQueue.java

...er/src/main/java/com/netease/arctic/server/optimizing/maintainer/IcebergTableMaintainer.java

ams/server/src/main/java/com/netease/arctic/server/table/DefaultTableService.java

ams/server/src/main/resources/mysql/upgrade-0.5.0-to-0.5.1.sql

core/src/main/java/com/netease/arctic/AmoroCatalog.java

dist/src/main/arctic-bin/conf/config.yaml

… paimon-integration-copy

...er/src/main/java/com/netease/arctic/server/optimizing/maintainer/IcebergTableMaintainer.java

...rver/src/main/java/com/netease/arctic/server/optimizing/maintainer/MixedTableMaintainer.java

…copy # Conflicts: # ams/server/src/main/java/com/netease/arctic/server/table/executor/OrphanFilesCleaningExecutor.java

…copy # Conflicts: # ams/server/src/main/java/com/netease/arctic/server/catalog/MixedIcebergCatalogImpl.java # ams/server/src/main/java/com/netease/arctic/server/dashboard/ServerTableDescriptor.java # ams/server/src/main/java/com/netease/arctic/server/table/executor/OrphanFilesCleaningExecutor.java # ams/server/src/main/java/com/netease/arctic/server/table/executor/SnapshotsExpiringExecutor.java # ams/server/src/main/java/com/netease/arctic/server/table/executor/TableRuntimeRefreshExecutor.java # core/src/main/java/com/netease/arctic/formats/mixed/MixedHiveTable.java

core/pom.xml

core/src/main/java/com/netease/arctic/AlreadyExistsException.java

core/src/main/java/com/netease/arctic/DatabaseNotEmptyException.java

core/src/main/java/com/netease/arctic/NoSuchTableException.java

core/src/main/java/com/netease/arctic/UnifiedCatalog.java

core/src/main/java/com/netease/arctic/formats/iceberg/IcebergSnapshot.java

core/src/main/java/com/netease/arctic/formats/iceberg/IcebergTable.java

zhoujinsong

I left some commnets, PTAL.

core/src/main/java/com/netease/arctic/formats/paimon/PaimonCatalogFactory.java

pom.xml

ams/server/src/main/java/com/netease/arctic/server/dashboard/FormatTableDescriptor.java

ams/server/src/main/java/com/netease/arctic/server/dashboard/ServerTableDescriptor.java

ams/server/src/main/java/com/netease/arctic/server/optimizing/maintainer/TableMaintainer.java

ams/server/src/main/java/com/netease/arctic/server/catalog/PaimonServerCatalog.java

ams/server/src/main/java/com/netease/arctic/server/dashboard/PaimonTableDescriptor.java

…copy

…copy # Conflicts: # pom.xml

ams/server/src/main/java/com/netease/arctic/server/dashboard/FormatTableDescriptor.java

...server/src/main/java/com/netease/arctic/server/dashboard/MixedAndIcebergTableDescriptor.java

ams/server/src/main/java/com/netease/arctic/server/dashboard/PaimonTableDescriptor.java

ams/server/src/main/java/com/netease/arctic/server/optimizing/maintainer/TableMaintainer.java

...rver/src/main/java/com/netease/arctic/server/optimizing/maintainer/MixedTableMaintainer.java

zhoujinsong · 2023-10-09T09:13:54Z

...server/src/main/java/com/netease/arctic/server/table/executor/SnapshotsExpiringExecutor.java

@@ -80,7 +47,10 @@ protected long getNextExecutingTime(TableRuntime tableRuntime) {

  @Override
  protected boolean enabled(TableRuntime tableRuntime) {
-    return tableRuntime.getTableConfiguration().isExpireSnapshotEnabled();
+    return tableRuntime.getFormat() == TableFormat.ICEBERG &&


Does all the Paimon format tables not require services provided by TableExecutor? If so, can we move the filter logic at a higher level?

Executor is a fundamental mechanism, and it's difficult to guarantee that Paimon won't use it in the future

ams/server/src/main/resources/mysql/upgrade-0.5.0-to-0.5.1.sql

core/src/test/java/com/netease/arctic/formats/IcebergCatalogTest.java

core/src/main/java/com/netease/arctic/formats/paimon/PaimonCatalogFactory.java

…ixedAndIcebergTableDescriptor.java Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>

…aimonTableDescriptor.java Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>

zhoujinsong

LGTM.
Thanks a lot for your work!

…format integration (apache#1960) * Integration paimon * Integration paimon * Integration paimon * Integration paimon * tmp * Support registering for the Paimon Catalog * Support registering for the Paimon Catalog * Support registering for the Paimon Catalog * Support registering for the Paimon Catalog * Fix ut * Polish code * Fix compile error * Merge master * polish code * Merge master * Fix UT * Polish code * Polish code * polish code * polish code * Update ams/server/src/main/java/com/netease/arctic/server/dashboard/MixedAndIcebergTableDescriptor.java Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * Update ams/server/src/main/java/com/netease/arctic/server/dashboard/PaimonTableDescriptor.java Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * tmp * polish code * polish code --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>

Integration paimon

d2ca938

github-actions bot added module:core Core module type:build module:ams-dashboard Ams dashboard module labels Sep 12, 2023

shidayang requested review from baiyangtx, HuangFru and zhoujinsong September 12, 2023 12:46

shidayang added 3 commits September 12, 2023 21:04

Integration paimon

59be5cd

Integration paimon

4639621

Integration paimon

06926dc

baiyangtx reviewed Sep 13, 2023

View reviewed changes

shidayang added 3 commits September 13, 2023 14:23

tmp

0df88f7

Support registering for the Paimon Catalog

ecaa102

Merge branch 'master' into paimon-integration-copy

ebe4b7e

baiyangtx mentioned this pull request Sep 14, 2023

[AMORO-1871]Support register paimon catalog #1975

Merged

3 tasks

shidayang added 3 commits September 14, 2023 15:07

Support registering for the Paimon Catalog

0c6b09a

Merge remote-tracking branch 'shidayang/paimon-integration-copy' into…

da3a94c

… paimon-integration-copy

Merge branch 'master' into paimon-integration-copy

5f66917

shidayang changed the title ~~Refactor code of "Table" and "Catalog" to support Paimon format integration~~ [AMORO-1906]Refactor code of "Table" and "Catalog" to support Paimon format integration Sep 14, 2023

shidayang added 8 commits September 14, 2023 15:36

Support registering for the Paimon Catalog

4bb61cf

Merge remote-tracking branch 'shidayang/paimon-integration-copy' into…

55de2ae

… paimon-integration-copy

Support registering for the Paimon Catalog

2b37a6e

Fix ut

cb0c241

Polish code

587d23f

Merge branch 'master' into paimon-integration-copy

bde9ec9

Fix compile error

9720442

Merge remote-tracking branch 'shidayang/paimon-integration-copy' into…

80e9a75

… paimon-integration-copy

shidayang requested a review from majin1102 September 15, 2023 03:24

HuangFru reviewed Sep 18, 2023

View reviewed changes

shidayang added 7 commits September 19, 2023 18:53

Merge remote-tracking branch 'origin/master' into paimon-integration-…

f1df45a

…copy # Conflicts: # ams/server/src/main/java/com/netease/arctic/server/table/executor/OrphanFilesCleaningExecutor.java

Merge master

ee9c632

polish code

b3df91a

Merge master

2aea58f

Fix UT

2b8088f

Merge branch 'master' into paimon-integration-copy

7ec198e

zhoujinsong reviewed Sep 21, 2023

View reviewed changes

shidayang added 3 commits September 25, 2023 14:44

Polish code

722f0ff

Merge branch 'master' into paimon-integration-copy

2686ef3

Polish code

cfc36ce

zhoujinsong reviewed Sep 25, 2023

View reviewed changes

polish code

52c7923

github-actions bot added the module:mixed-hive Hive moduel for Mixed Format label Oct 9, 2023

shidayang and others added 4 commits October 9, 2023 15:40

Merge remote-tracking branch 'origin/master' into paimon-integration-…

3e5982e

…copy

Merge remote-tracking branch 'origin/master' into paimon-integration-…

d43afd1

…copy # Conflicts: # pom.xml

polish code

4c5457a

Merge branch 'master' into paimon-integration-copy

d8b769e

zhoujinsong reviewed Oct 9, 2023

View reviewed changes

shidayang and others added 5 commits October 9, 2023 19:27

Update ams/server/src/main/java/com/netease/arctic/server/dashboard/M…

661b136

…ixedAndIcebergTableDescriptor.java Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>

Update ams/server/src/main/java/com/netease/arctic/server/dashboard/P…

2306f02

…aimonTableDescriptor.java Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>

tmp

5f87798

polish code

0b2c393

polish code

a5beb36

github-actions bot added the module:ams-server Ams server module label Oct 9, 2023

zhoujinsong approved these changes Oct 10, 2023

View reviewed changes

zhoujinsong merged commit 4ae100a into apache:master Oct 10, 2023
5 of 6 checks passed

XBaith mentioned this pull request Nov 20, 2023

[AMORO-1960] Refactor data expiration to TableMaintainer #2328

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMORO-1906]Refactor code of "Table" and "Catalog" to support Paimon format integration #1960

[AMORO-1906]Refactor code of "Table" and "Catalog" to support Paimon format integration #1960

shidayang commented Sep 12, 2023

codecov bot commented Sep 13, 2023 •

edited

Loading

zhoujinsong left a comment

zhoujinsong Oct 9, 2023

shidayang Oct 9, 2023

zhoujinsong left a comment

[AMORO-1906]Refactor code of "Table" and "Catalog" to support Paimon format integration #1960

[AMORO-1906]Refactor code of "Table" and "Catalog" to support Paimon format integration #1960

Conversation

shidayang commented Sep 12, 2023

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

codecov bot commented Sep 13, 2023 • edited Loading

Codecov Report

zhoujinsong left a comment

Choose a reason for hiding this comment

zhoujinsong Oct 9, 2023

Choose a reason for hiding this comment

shidayang Oct 9, 2023

Choose a reason for hiding this comment

zhoujinsong left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 13, 2023 •

edited

Loading